Pgvector port - #182
Conversation
| PINECONE_ENVIRONMENT=environment | ||
| PINECONE_KEY=key | ||
|
|
||
| # POSTGRE |
| replicate==0.18.1 No newline at end of file | ||
| replicate==0.18.1 | ||
| psycopg[binary] | ||
| psycopg2-binary No newline at end of file |
| PINECONE_KEY=key | ||
|
|
||
| # POSTGRE | ||
| POSTGRE_HOST=localhost |
| get_postgre_cursor().execute('SELECT * FROM embedding ORDER BY values <-> %s LIMIT 50', (np.array(query_vector),)) | ||
| query_matches = get_postgre_cursor().fetchall() | ||
|
|
||
| # query( |
| used_messages = list( | ||
| filter( | ||
| lambda match: match["id"] | ||
| lambda match: match[2] |
There was a problem hiding this comment.
Move magic number to constant
| gunicorn==20.1.0 | ||
| replicate==0.18.1 No newline at end of file | ||
| replicate==0.18.1 | ||
| psycopg[binary] |
| from flask import Flask, request, jsonify | ||
| from .config import get_google_tasks_service_account | ||
| from .external_services.pinecone import get_pinecone_index | ||
| from .external_services.postgre_vector import get_postgre_cursor |
YuraLukashik
left a comment
YuraLukashik
left a comment
There was a problem hiding this comment.
There should be a way to switch between the existing implementation with Pinecone and the new one with Postgres. Some level of abstraction is needed for that.
| *** | ||
|
|
||
|
|
There was a problem hiding this comment.
What is that space for?
There was a problem hiding this comment.
let me remove one line. is it ok?
There was a problem hiding this comment.
Why do we need these 3 new lines?
There was a problem hiding this comment.
Ah this is my mistake, let me remove them.
| register_vector(cur) | ||
|
|
||
| # cur.execute('DROP TABLE IF EXISTS embedding') | ||
| cur.execute('CREATE TABLE IF NOT EXISTS embedding (id bigserial PRIMARY KEY, namespace text, chunk_id text, metadata text, values vector)') |
There was a problem hiding this comment.
Should we really do that during every connection?
There was a problem hiding this comment.
As you know app will not create table if there is already exist. Let me find best way
|
|
||
| conn = None | ||
| try: | ||
| conn = psycopg2.connect( |
There was a problem hiding this comment.
We should create a connection only when it's needed, not when Python modules are loaded.
|
|
||
|
|
||
| def delete_pinecone_embedding(embeddings: list[Embedding], pinecone_index, pinecone_namespace): | ||
| def delete_db_embedding(embeddings: List[Embedding], postgres_cursor, namespace): |
There was a problem hiding this comment.
Why don't we make it abstract?
| is_actual_message, \ | ||
| slack_names_map, filter_messages, load_previous_messages, load_subsequent_messages | ||
|
|
||
| import numpy as np |
There was a problem hiding this comment.
Not used in this file.
3 participants
No description provided.